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LONG-TERM  GOALS 

This  project  seeks  to  develop  the  technology  to  fuse  high  spatial  resolution  MultiSpectral  Imagery 
(MSI)  with  lower  spatial  resolution,  but  higher  spectral  resolution,  HyperSpectral  Imagery  (HSI)  to 
provided  enhanced  target  detection  and  battlespace  characterization.  It  seeks  to  push  beyond 
traditional  PAN  sharpening  techniques  to  develop  applications  that  will  increase  the  accuracy  and 
fidelity  of  the  sharpened  spectral  imagery. 

OBJECTIVES 

1)  Co-register  high  resolution  MSI  data  with  HSI  data  collected  during  the  Fall  of  2004. 

2)  Correct  the  MSI  data  for  illumination  and  atmospheric  interference  to  derive  a  broadband  remote 
sensing  reflectance,  Rrs,  measurement  from  the  MSI  data. 

3)  Develop  methodologies  to  fuse  the  HSI  data  with  the  higher  spatial  resolution  MSI  data,  using  the 
spectral  characterization  of  the  MSI  Rrs  data  stream. 

APPROACH 

The  fusion  of  remote  sensing  data  from  different  sensors  has  a  long  history  of  success  in  the  terrestrial 
environment  (Pohl  and  Genderen,  1998).  Its  application  in  ocean  remote  sensing  has  accelerated  in 
recent  years  with  the  use  of  multiple  resolution,  multiple  frequency  optical  and  microwave  imaging. 
The  optical  imagers  include  both  active  (e.g.  LIDAR)  and  passive  (e.g.  HyperSpectral  Imagers,  HSI). 
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IMPACTS/APPLICATIONS 


The  abilities  to  detect  targets  and  characterize  the  environment  in  the  Very  Shallow 
Water  (VSW)  and  beach  zone  are  critical  to  MIW,  MCM,  and  NSW  operations.  The 
image  fusion  techniques  here  will  provide  enhanced  abilities  to  detect  and  characterize 
target  in  access  denied  areas  using  both  geomorphology  (shape)  and  spectral  (color) 
signatures.  In  addition,  the  successful  development  of  these  techniques  will  feed  back 
into  the  development  of  new  imaging  systems  for  space  and  aircraft  platforms,  including 
Unmanned  Aircraft  Systems  (UAS). 

This  work  demonstrated  the  ability  to  fuse  high  resolution  multispectral  imagery  with 
lower  resolution  hyperspectral  imagery  using  a  new  method  of  color  tinting  fusion.  On 
simulated  multispectral  imagery  we  achieved  a  ~6%  reconstruction  error  across  the 
VNIR  spectrum;  on  actual  multispectral  imagery  we  achieved  ~20%  reconstruction  error. 
The  difference  between  the  simulated  and  actual  was  mostly  attributed  to  the  inability  to 
calibrate  and  atmospherically  correct  the  multispectral  imagery. 


Microwave  imagers  are  typically  Synthetic  Aperture  Radar  (SAR).  While  the  field  has  been  generally 
confined  to  airborne  and  space-based  platforms,  imagery  data  fusion  may  also  include  some 
combination  of  data  from  these  platforms  with  data  from  in-water  systems  such  as  multibeam  and  side- 
scan  sonar  imagery.  In  short,  data  fusion  is  the  process  by  which  two  or  more  streams  of  data  of 
different  temporal  and  spatial  resolution  are  combined  to  produce  synthesized  products  which  are 
unattainable  from  a  discrete  image. 

The  advantages  of  data  fusion  are  fairly  clear.  In  the  temporal  domain,  data  fusion  allows  for  the  more 
complete  imaging  of  an  area  intermittently  covered  by  clouds.  It  may  also  provide  better  analysis  of 
movement  or  time-dependent  change  in  targets  and/or  environmental  conditions.  In  the 
electromagnetic  frequency  domain,  targets  which  are  visible  in  one  image  type  may  be  less  than  visible 
in  another,  and  vice  versa.  SAR  imagery  may  illuminate  target  elevations  or  directional  wave  spectra, 
whereas  optical  imagery  may  identify  targets  by  shape  and  color.  Combining  the  targeting  and 
identification  of  these  different  data  types  provides  target  recognition  and  environmental 
characterization  with  high  confidence  and  lower  false  alarm  rates  than  may  be  found  from  single 
source  imagery.  In  the  spatial  domain,  objects  that  are  identifiable  on  the  basis  of  size  and  shape  in 
panchromatic  (PAN)  imagery  are  often  unidentifiable  by  their  geometric  dimensions  in  lower 
resolution  multispectral  or  hyperspectral  imagery.  The  converse  is  that  the  spectral  imagery  provides 
the  ability  to  identify  targets  or  characterize  the  environment  based  on  unique  spectral  signatures, 
which  is  unavailable  in  the  panchromatic  imagery,  but  may  not  be  able  to  resolve  the  shape  or 
dimensions  of  small  targets. 

This  project  seeks  to  co-locate  simultaneously  collected  MSI/HSI  data  streams  to  develop  enhanced 
imagery  sharpening  capabilities  for  use  in  target  detection  and  environmental  characterization.  It  seeks 
to  build  upon  PAN  sharpening  techniques  by  using  the  full  spectral  information  provided  by  MSI  data 
streams.  We  hypothesize  that  the  best  way  to  attempt  this  image  fusion  is  at  the  level  of  remote 
sensing  reflectance,  so  the  MSI  data  must  be  corrected  for  illumination  and  atmospheric  interference. 
The  spectral  information  from  the  co-registered  MSI/HSI  Rrs  data  will  then  be  fused  to  provide  more 
accurately  sharpened  HSI  data  for  spectral  target  algorithms. 

WORK  COMPLETED  -  Year  1 

This  project  began  in  April  of  2005,  and  started  with  data  previously  collected  by  FERI.  FERI,  in 
collaboration  with  the  California  State  University  System,  under  the  NOAA-funded  California  Center 
for  Integrative  Coastal  Observation,  Research  and  Education  (CICORE)  Program 
(http://www.feriweb.org/proiects/ci-core/)  recently  (Fall  2004)  collected  over  5,000  square  kilometers 
of  0.3  meter  MSI  and  3  meter  HSI  data  over  a  wide  variety  of  coastal  environments  along  the 
California  coast  ( http  ://www .  feri  web  .or  g/proi  ects/ci-core/arcims/) .  The  survey  sites  include  Humbolt 
Bay,  San  Francisco  Bay,  Monterey  Bay,  the  Big  Sur  coast,  San  Luis  Bay,  Santa  Barbara,  Newport,  and 
San  Diego  Harbor.  The  MSI  instrumentation  was  provided  under  a  cooperative  agreement  with  the 
Applanix  Systems  Integration  Group  (ASIG),  a  subsidiary  of  the  Trimble  Corporation.  This  MSI 
system  (Digital  Sensor  System;  http://www.applanix.com/products/dss  index.php)  was  provided  as 
part  of  the  Applanix  Position  and  Orientation  System  (POS  AV; 

http://www.applanix.com/products/posav  index.php).  which  was  being  evaluated  as  a  replacement  for 
the  current  POS  system  on  the  NRL  PHILLS  2  airborne  hyperspectral  imager. 

The  raw  DSS  imagery  (MSI  imagery)  and  POS  data  was  provided  to  FERI  by  ASIG  after  the  flights. 
The  processing  and  geo-positioning  of  the  MSI  data  was  completed  by  FERI  under  this  ONR  project. 
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This  geo-positioning  system  is  of  much  greater  accuracy  than  the  previous  POS  system  on  the  PH  ILLS 
2.  The  processing  of  this  POS  data  stream  required  additional  training,  which  was  provided  by  ASIG. 
In  addition,  the  DSS  is  a  frame  camera  imagery,  which  is  very  different  from  the  line  scanning  imager, 
which  is  at  the  heart  of  the  PHILLS  2.  The  processing  of  the  MSI  data  stream  itself  required  additional 
training  in  the  photogrammetry  and  ortho-rectification.  The  training  was  completed,  as  well  as  the 
processing  of  the  raw  POS  data  and  DSS  imagery  to  ortho-rectified  (Figure  1 ).  These  data  are  to  be 
used  with  the  co-registered  HSI  data  from  the  Morro  Bay  region  (Figure  2). 


Figure  1.  FERI  MSI  of  selected  region  in  Morro  Bay,  CA  collected  November  11,  2004.  This  area 
has  been  chosen  as  a  focus  site  for  the  development  of  the  MSI/HSI  fusion  algorithms.  Note  the 
enhanced  spatial  resolution  when  compared  to  the  HSI  imagery  in  Figure  2. 

RESULTS  -  Year  1 

The  MSI/HSI  data  has  been  co-registered  in  anticipation  of  the  next  step  in  the  development  of  the 
spectral  fusion  over  the  next  6  months.  One  issue  that  has  been  found  during  the  training  of  FERI 
personnel  by  ASIG,  which  may  be  cause  for  difficulties,  is  that  the  radiometric  and  spectral 
calibrations  of  the  DSS  are  not  what  have  been  purported  by  the  manufacturer.  The  calibration  issues 
need  to  be  resolved  prior  to  the  development  of  the  Rrs  fusion  approach,  for  it  will  be  very  difficult  to 
correct  for  illumination  and  atmospheric  effects  without  accurate  calibration.  This  recalibration  of  the 
MSI  sensor  by  FERI  personnel  is  scheduled  to  occur  this  fall  (funded  by  Trimble),  so  these  data  should 
be  available  later  this  year  to  complete  the  initial  development  of  the  Rrs  fusion  approach. 
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Figure  2.  FERI HSI  of  selected  region  in  Morro  Bay,  CA  collected  November  11,  2004.  This  area 
has  been  chosen  as  a  focus  site  for  the  development  of  the  MSI/HSI  fusion  algorithms.  Note  the 
reduced  spatial  resolution  when  compared  to  the  MSI  imagery  in  Figure  1. 

WORK  COMPLETED  -  Year  2 

The  work  during  this  period  looks  at  exploiting  the  information  available  from  multiple  sensors  aboard 
an  aircraft.  Particularly,  we  wish  to  combine  information  from  a  high  spatial  but  low  spectral 
resolution  camera  and  a  low  spatial  but  high  spectral  resolution  camera.  The  desired  final  image 
should  have  both,  high  spatial  and  high  spectral  resolution.  Data  fusion  can  be  defined  as  the  fusion  of 
complementary  information  from  multiple  sources.  In  this  work,  we  focus  on  the  combination  of  low- 
resolution  multi-spectral  data  with  high-resolution  tri-band  (RGB)  data.  Many  image  fusion 
techniques  have  been  proposed  to  merge  multi-spectral  (MS)  and  panchromatic  (Pan)  images,  with 
complementary  characteristics  of  spatial  and  spectral  resolutions.  They  are  mostly  component 
substitution  based  methods  such  as  IHS,  Brovey  and  MULTI. 

The  IHS  (Intensity,  Hue,  and  Saturation)  transformation  is  mostly  used  for  three  band  images.  In  the 
IHS  transformation,  the  difference  between  the  high  resolution  and  low  resolution  intensity  images  is 
added  to  the  up-sampled  MS  bands.  However,  as  the  spectral  response  of  the  two  cameras  is  usually 
different,  this  procedure  is  prone  to  spectral  distortion.  When  more  than  three  bands  are  present,  this 
transformation  can  be  applied  to  groups  of  three  consecutive  bands  at  a  time.  Alternatively,  a  PCA 
(Principal  Component  Analysis)  approach  can  be  used. 
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The  Brovey  fusion  method  computes  the  ratio  between  a  panchromatic  cell  value  and  the  average  of 
the  corresponding  multi-spectral  cell  values  and  uses  that  ratio  to  compute  the  final  color  component 
values  for  the  pan-sharpened  image.  The  Gram-Schmidt  (GS)  technique  is  another  component 
substitution  technique  for  spectral  sharpening.  In  the  GS  method,  the  low-resolution  version  of  the 
PAN  image  is  obtained  as  the  pixel  average  of  the  MS  bands.  However,  as  discussed  earlier,  because 
of  the  radiometric  differences  between  the  two  sensors,  the  reconstructed  image  is  prone  to  suffer  from 
spectral  distortion.  The  purpose  of  this  work  is  to  mitigate  this  distortion  by  reconciling  the  two 
sensors  radiometrically,  prior  to  performing  the  GS  fusion.  A  similar  approach  is  taken  in  Aiazzi  et  al, 
2006  where  regression  coefficients  are  determined  to  reconcile  the  two  sensors  prior  to  GS  sharpening. 
In  our  case,  however,  the  sensor  response  is  known  a  priori,  obviating  the  need  for  determining 
coefficients. 

RESULTS  -  Year  2 

The  purpose  of  this  work  is  to  construct  a  high  spatial  and  high  spectral  resolution  image  using  images 
from  two  sensors.  One  sensor  is  a  high  spatial  resolution  frame  camera  -  the  DSS-322.  The  other  is  a 
high  spectral  resolution  camera,  FERI’s  SAMSON.  The  Gram  Schmidt  procedure  described  above 
was  used  to  combine  information  from  the  two  cameras  to  obtain  a  high  spectral,  high  spatial 
resolution  image. 

To  evaluate  the  validity  of  the  reconstructed  image,  the  hyper-spectral  and  multi-spectral  images  had  to 
be  down-sampled  spatially  and  then  used  to  reconstruct  the  original  hyper-spectral  image.  The  error 
between  the  reconstructed  and  original  hyper-spectral  image  is  a  quantitative  indicator  of  the  proposed 
procedure’s  validity. 

Image  fusion  was  studied  under  two  conditions  -  practical  and  ideal.  In  the  practical  scenario,  actual 
hyper-spectral  and  multi-spectral  images  were  used.  For  the  ideal  scenario,  an  actual  hyper-spectral 
and  a  simulated  multi-spectral  image  (simulated  using  the  hyper-spectral  image)  were  used.  By  using 
hyper-spectral  and  multi-spectral  images  derived  from  the  same  sensor,  many  error  sources  such  as 
registration,  calibration  and  atmosphere  are  eliminated.  Comparing  results  from  these  two  scenarios 
helps  quantify  the  error  contribution  from  these  sources. 

The  simulated  DSS  image  was  obtained  from  the  hyper-spectral  image  using  the  spectral  response 
function  of  the  DSS  camera.  Each  band  of  the  DSS  camera  was  simulated  by  taking  a  dot  product 
between  that  band’s  spectral  response  function  with  the  actual  wavelength  responses  from  the  hyper- 
spectral  signal  as  shown  in  Equation  1 . 

DSS(i,j,k)  =  YjHYP(i,j,A)xSRFk(A)  (1) 

'l='U 

Here,  k  indicates  the  DSS’s  band,  k=  { 1 ,2,3 } ,  (ij)  indicate  the  pixel  location,  SRFk  (X)  is  the  klh  band’s 
response  at  wavelength  X  . 
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Two  approaches  to  image  fusion  were  compared.  In  the  first  approach,  GS  was  run  using  the  hyper- 
spectral  image  and  one  of  the  DSS  bands  to  obtain  the  reconstructed  hyper-spectral  image  as  shown  in 
Figure  3. 


Hyper-spectral  Multispectral 


Figure  3.  In  the  traditional  Gram-Schmidt  approach  a  single  PAN  channel  is  used  to  fuse  the  high 
spatial  resolution  with  the  lower  spatial ,  but  high  spectral  resolution  image. 

In  the  second  approach,  the  hyper-spectral  image  was  multiplied  by  each  band’s  SRF  response,  to 
obtain  three  ‘tinted’  hyper-spectral  images.  Three  reconstructions  were  performed  to  obtain  ‘tinted’ 
reconstructed  images.  They  were  then  multiplied  by  the  corresponding  inverse  SRF  response  to  obtain 
three  versions  of  the  original  hyper-spectral  image.  Figure  2  illustrates  how  a  hyper-spectral  image 
was  generated  using  the  red  channel. 


Inverse  Red  Filter 


Figure  4.  In  the  modified  Gram-Schmidt  approach  each  RGB  channel  is  used  only  with  the 

overlapping  bands  in  the  HSI  image  cube. 
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Tinting  the  image  prior  to  using  the  GS,  ensures  that  the  hyper-spectral  and  multi-spectral  images  have 
similar  spectral  content  leading  to  better  reconstruction.  Once  three  versions  of  the  reconstructed 
hyper- spectral  image  are  obtained,  they  can  be  averaged  to  obtain  a  final  reconstructed  version.  In  this 
work,  a  more  elegant  fusion  scheme  was  used  as  shown  in  Equation  2. 

HYPavg(  /l  )=Wr(  A  )HYPr(  A  )+Wg(  A  )HYPg(  A  )+Wb(  A  )HYPb(  A )  (2) 

Where  Wr(  * )  =  SRFr(  * )  /  (SRFr(  * )  +  SRFg(  * )  +  SRFb(  * )).  This  means  that  instead  of  weighting 
the  three  images  equally  (as  you  would  in  an  average),  they  are  now  weighted  in  proportion  to  their 
dominance  at  each  wavelength. 

DISCUSSION  -  Year  2 

In  this  section,  we  compare  the  performance  of  the  modified  method  to  that  of  the  original  method  in 
the  ideal  scenario.  Then,  we  compare  our  method’s  performance  in  the  two  scenarios  (ideal  and 
practical).  This  is  followed  by  a  detailed  discussion  of  the  results. 

First  we  evaluate  the  efficacies  of  the  proposed  modified  method  and  the  original  method  in  the  ideal 
scenario.  Figure  3  and  Figure  4  described  the  workings  of  the  two  methods.  A  three  band  multi- 
spectral  image  was  simulated  from  the  hyper-spectral  image  as  described  by  Equation  1 .  The  original 
hyperspectral  image  was  reconstructed  using  both  methods  and  the  errors  were  evaluated. 

To  characterize  the  nature  of  the  reconstruction  error,  we  studied  its  distribution  along  wavelengths 
and  magnitudes.  The  error  is  expressed  as  a  fraction  of  the  original  hyper-spectral  image. 

Figure  5  shows  the  reconstruction  errors  in  the  ideal  scenario  using  both  methods.  The  X-axis  ranges 
from  band  #40  to  band  #80  spanning  wavelengths  from  400nm  to  700nm.  The  Y-axis  expresses  the 
error  as  a  fraction  of  the  original  value.  From  (a),  it  is  evident  that  the  reconstruction  error  using 
Method  1  is  in  the  1 5-20%  range.  From  (b),  it  can  be  seen  that  using  the  Tinting  method,  the  error 
drops  to  the  6-8%  range. 
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Figure  5.  The  graph  on  the  left  shows  the  errors  in  the  ideal  comparison  case  using  the  traditional 
Gram-Schmidt  approach,  and  the  range  is  ~15-20%  over  the  spectrum.  In  the  modified  color 
tinted,  Gram-Schmidt  approach  (right  graph)  there  is  a  marked  reduction  in  the  reconstruction 

error,  range  is  ~6-8%  over  the  spectrum. 


MSI  (DSS  RGB)  HSI  Fused  MSI/HSI 


Figure  6.  The  color  tinted  fusion  of  high  spatial  resolution  MSI  (left  figure)  with  lower  spatial 
resolution  HSI  (middle  figure)  into  a  high  spatial  resolution,  high  spectral  resolution  image  (right 
image).  The  bottom  row  of  images  represents  the  spectral  plots  at  the  pixel  located  at  the  center  of 

the  red  cross  hairs  in  the  images  directly  above  them. 

Next  we  compare  the  reconstruction  errors  in  two  different  scenarios  using  the  Tinting  method.  Figure 
7  compares  the  errors  in  the  ideal  and  practical  scenarios. 
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Figure  7.  This  graph  shows  the  reconstruction  error  in  the  practical  case,  where  the  DSS  imagery  is 
used  in  the  modified  Gram-Schmidt  approach  to  fusing  the  MSI/HSI  data.  The  increase  in  the 
error  range  to  -22%  results  from  differences  in  (1)  the  unknown  calibration  of  the  DSS,  (2)  view 
angle  and  its  impacts  on  atmospheric  and  illumination  contamination,  and  (3)  pixel  co-registration 

errors. 

From  Figure  7,  it  is  evident  that  as  we  move  from  the  Ideal  to  the  Practical  scenario,  the  average  error 
jumps  from  about  5%  to  22.5%.  There  are  multiple  reasons  for  this  jump  in  reconstruction  error  as  we 
move  from  the  Ideal  to  Practical  scenarios. 

First,  errors  arise  because  the  multi-spectral  camera  is  not  accurately  calibrated.  We  have  sought  to 
work  with  the  manufacturer  to  resolve  these  calibration  issues,  but  as  of  yet,  this  sensor  can  not  be 
calibrated  to  any  radiometric  standard. 

Second,  the  two  sensors  look  at  the  same  location  through  different  view  angles,  the  hyper-spectral 
camera  view  is  nominally  at  a  nadir  angle  in  the  along  track  direction,  and  ~+/-20  degrees  in  the  cross 
track  direction  (pushbroom  type  sensor),  whereas  the  multi-spectral  camera  usually  looks  at  an  angle 
varying  from  0-20  degrees  in  both  the  along  track  and  cross  track  directions  (framing  type  sensor). 

Thus  their  path  lengths  through  the  atmosphere  and  each  pixels  relative  illumination  are  different, 
which  yield  a  different  overall  spectral  signal  between  the  DSS  and  SAMSON  when  the  data  are  not 
corrected  for  atmospheric  and  illumination  differences. 

Third,  since  two  cameras  have  different  resolution  and  lens  aberration,  image  registration  is  inaccurate. 
It  was  observed  that  a  registration  error  of  one  pixel  can  contribute  up  to  5%  error. 

We  hypothesize  that  of  the  20%  average  error,  errors  due  to  atmosphere  and  calibration  account  for 
10%  of  the  error,  5%  due  to  image  mis-registration  and  the  remaining  5%  is  inherent  in  the  GS 
procedure.  In  spite  of  the  errors,  this  approach  shows  great  promise  in  being  able  to  generate  HSI  data 
at  the  spatial  resolution  of  PAN  and  RGB  imagers.  We  need  to  cross  the  critical  step  of  calibrated  the 
RGB/MSI  data  so  that  we  may  apply  this  technique  to  the  reflectance  data  itself,  rather  than  the  sensor 
radiance  data.  This  would  greatly  reduce  the  errors  in  the  Practical  Approach  shown  here,  allowing  us 
to  approach  our  5%  reconstruction  error  seen  in  the  Ideal  Approach. 

IMPACTS/APPLICATIONS 

The  abilities  to  detect  targets  and  characterize  the  environment  in  the  Very  Shallow  Water  (VSW)  and 
beach  zone  are  critical  to  MIW,  MCM,  and  NSW  operations.  The  image  fusion  techniques  here  will 
provide  enhanced  abilities  to  detect  and  characterize  target  in  access  denied  areas  using  both 
geomorphology  (shape)  and  spectral  (color)  signatures.  In  addition,  the  successful  development  of 
these  techniques  will  feed  back  into  the  development  of  new  imaging  systems  for  space  and  aircraft 
platforms,  including  Unmanned  Aircraft  Systems  (UAS). 

This  work  demonstrated  the  ability  to  fuse  high  resolution  multispectral  imagery  with  lower  resolution 
hyperspectral  imagery  using  a  new  method  of  color  tinting  fusion.  On  simulated  multispectral  imagery 
we  achieved  a  ~6%  reconstruction  error  across  the  VNIR  spectrum;  on  actual  multispectral  imagery  we 
achieved  —20%  reconstruction  error.  The  difference  between  the  simulated  and  actual  was  mostly 
attributed  to  the  inability  to  calibrate  and  atmospherically  correct  the  multispectral  imagery. 
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